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ABSTRACT 

An Earth science professional-development program for in-service middle- and high-school teachers increased participants' 
content knowledge about weather, climate, and climate change in addition to increasing their confidence in their assessment 
and teaching skills. The curriculum and funding gave them time, funding, and the help of content and pedagogic experts to 
build and test new lesson plans on these topics. During the summer workshop, the teachers participated in field trips, hands- 
on laboratory experiences, and cooperative inquiry projects. They studied weather and mechanisms that determine climate 
during the first week. The second week focused on geologic time and changes in the Earth system that drive climate changes 
on long timescales. The major topic for the third week was anthropogenic climate change. During the workshop, the teachers 
designed lesson plans for multiday units on workshop topics. The teachers took a content pretest at the start of the summer 
workshop and a post-test at the end. Their average score increased from 38.1% to 75.6%. The follow-up program included 
three meetings and two classroom visits by a veteran teacher during the subsequent school year. The participants taught their 
lesson plans and administered pre- and post-tests that they had designed to their own students. Their class scores showed 
normalized gains ranging from 16% to 88%. Surveys taken at the beginning of the summer workshop and the end of the 
follow-up program indicated an average increase in their confidence of their ability to teach science and in their willingness to 
employ hands-on, cooperative, and inquiry learning techniques in their classrooms. © 2014 National Association of Geoscience 
Teachers. [DOI: 10.5408/13-039.1] 
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INTRODUCTION 

Eighteen in-service teachers (mostly middle and high 
school) completed the Wright State Institute for Teacher 
Quality (ITQ) Earth science summer workshop in 2011 on 
climate change over geologic time. Most finished a follow¬ 
up program during the 2011-2012 academic year. These 
experiences enhanced their content knowledge of climate 
change and Earth history and, in many cases, broadened 
their experience of inquiry learning. The Earth science 
workshop ran alongside a physical science and a life science 
workshop. Participants could only take part in one of the 
three workshops during any given year, but several of the 
teachers in the 2011 Earth science workshop had completed 
an ITQ life science workshop in previous years. 

Instructor and evaluator salaries, the cost of the 
participants' textbooks and the classroom materials for their 
unit plans (up to $425/participant), participant stipends 
(awarded at the end of the follow-up program), the costs for 
substitutes teachers for the three follow-up workshops, and 
other expenses were covered by a 2010 Improving Teacher 
Quality grant from the Ohio Board of Regents (OBR). 
Participants received graduate credit from the Department of 
Earth and Environmental Sciences for the summer workshop 
component and from the Department of Teacher Education 
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for the follow-up program. Wright State University waived 
the fees and tuition costs. The only costs to the teachers 
themselves were transportation to the workshop and most 
lunches. 

Ohio had recently adopted new science content 
standards (Ohio Department of Education, 2011a, 2011b). 
All three ITQ workshops offered opportunities to deepen 
content knowledge of topics relevant to the standards and to 
develop lesson plans that addressed them. This paper 
describes the new curriculum developed for the Earth 
science workshop, the effects on the participants and on 
their students, and some of the challenges to involving 
participants from high-needs districts in an effective teacher 
workshop. 

At a national level, the first recommendation in the 
report Rising above the Gathering Storm: Energizing and 
Employing America for a Brighter Economic Future (Committee 
on Prospering in the Global Economy of the 21st Century, 
2007) is to enhance the content knowledge of K-12 science, 
mathematics, and engineering teachers. Not only could 
these teachers increase the number of American scientists 
and engineers (over time), but they also have the potential to 
increase the science literacy, technological skills, and 
mathematical ability of all young Americans. The committee 
specifically recommends funding summer institutes for K-12 
teachers to develop greater science and math content 
knowledge with the goal of increasing student learning (A- 
2 Part 1) and funding competitive institutional grants for 
science and math master's degree programs (A-2 Part 2). 

The few scientific studies of teacher professional 
development examined by Yoon et al. (2007) indicate that 
effective programs generally take time (more than 14 contact 
hours), include a follow-up program after the workshop, and 
are taught directly to the teachers by researchers, as opposed 
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to indirect training using district personnel. The Summer 
Science and Math Institutes for Teacher Quality (ITQ) at 
Wright State University meet or exceed these recommenda¬ 
tions. This program has been run at least 10 times since 
1998, with funding from the Ohio Board of Regents 
Improving Teacher Quality State Grants Program, and it 
generally consists of two or three workshops running 
simultaneously, with about 24 participants apiece (Basista 
and Mathews, 2002). The workshops are taught by mixed 
teams of college professors with doctoral degrees and highly 
qualified, experienced K-12 teachers. The summer programs 
include at least 72 contact hours, rely on inquiry more than 
on didactic instruction, and focus on increasing participants' 
content knowledge and pedagogical content knowledge 
(PCK). They are assessed through pre- and post-tests of 
both the participants and the participants' students. These 
workshops offer continuing education credit and graduate 
credit towards a master's degree. 

Ohio had developed its own set of science content 
standards in 2003 (Ohio Department of Education, 2003), 
but these were replaced with a new set in 2011. Ohio 
teachers needed to update their content knowledge and 
rebuild their curricula to implement the new state science 
standards. Many of the topics for Earth science at the 
middle- and high-school levels are the same, but not all. The 
emphasis was shifted from topics to themes (called 
"strands") that connected life, Earth, and physical science 
at the middle-school level. For example, in 7th grade, the 
theme is cycles of matter and energy exchange. Inquiry and 
technological literacy are no longer separate standards but 
are integrated within the others (Ohio Department of 
Education, 2011a). The new high-school standards outline 
specific recommendations for courses in advanced sciences, 
including physical geology and environmental science (Ohio 
Department of Education, 2011b). 

At the middle-school level, Ohio's science standards are 
divided into three areas: Earth and space science, life science, 
and physical science, so the ITQ workshops for 2011 were 
divided into similar categories. The topic for the 2011 Earth 
science workshop was climate change over geologic and 
recent timescales. Climate change and geologic time are 
relevant to several of the Earth and space science and life 
science standards, but these topics are complex and 
controversial. Many teachers are not confident in their 
understanding of these topics and may avoid teaching them 
(Trend, 2001; Dahl et al., 2005; Wise, 2010), despite the 
pressure from the state and their administrations. 

Climatology and Earth history are great vehicles for 
teaching about the nature of science (as described by 
McComas and Almazroa, 1998). In Ohio, as in many areas, 
students can collect fossils of marine organisms far from the 
modern ocean, and these provide empirical evidence for sea- 
level change. New evidence can overthrow established 
theories, so every theory is somewhat tentative, even if well 
supported at the present time. For instance, careful analysis 
of glacial deposits in Europe and North America led 
scientists to infer that there had been four ice ages during 
the Pleistocene and that these could be correlated globally 
(e.g. Wright, 1914), but closer examination of those deposits 
(e.g., Richmond and Fullerton, 1986) and analysis of ls O 
ratios in ocean cores indicate that there were many more 
than four ice ages, and correlation, even across a few 
hundred miles, is still difficult. Scientific knowledge is 


socially negotiated (McComas and Almazroa, 1998), often 
through debates, like the ones over the cause of the 
extinction of the dinosaurs (Glen, 1994). 


PARTICIPANT POPULATION 

The ITQ program was funded by a grant from the OBR; 
the original proposal targeted teachers from two local high- 
needs districts, Dayton City and Trotwood. Administrators 
from these two partner districts sent letters of support for the 
program to the OBR. Teachers from Dayton City Schools 
were allowed to register a month before teachers from other 
districts, and two of the district's curriculum specialists were 
given details about the program to use for recruiting. 
However, only seven Dayton City teachers and one Trot¬ 
wood teacher applied to participate in the Earth science ITQ 
workshop. Of those, two dropped out of the workshop after 
the first week, and two did not complete the follow up- 
program. Most of these teachers worked in schools rated 
"Academic Watch" in 2010-2011 (Table I), meaning that 
their students have scored poorly on standardized tests for at 
least two consecutive years and that they are eligible for state 
sanctions. 

Anyone with a bachelor's degree who taught school in 
Ohio was eligible to participate in the ITQ program. Of the 
12 applicants from nontargeted districts, most taught in 
schools with an "Excellent" rating. One came from a private 
school (Table I). The Earth science workshop began with 20 
in-service teachers, 14 female and 6 male. Fourteen taught 
grades K-8, including two special-education teachers, and 
six taught high-school science (Table I). Data on participant 
race and age were not used to evaluate the project. 

The ITQ summer workshop included two other work¬ 
shops taught in nearby classrooms that followed the same 
format and used the same schedule: physical science and life 
science. The physical science workshop focused on energy, 
and the life science one focused on evolution. The 
participants were similar in background to those in the 
Earth science workshop, and the classes suffered similar 
dropout rates. The participants in the Earth science 
workshop worked with those in the life science workshop 
on a couple of projects during week 2. 

PROGRAM CURRICULUM 

The first week of the summer workshop focused on the 
causes of weather and climate, the second concentrated on 
geologic time and past climate change, and the third week 
addressed recent anthropogenic climate change (Supple¬ 
mentary Material 1). The participants and instructors met in 
a science classroom at Wright State University 4 days a week 
for 3 weeks in June 2011 (the Earth science schedule is in 
Supplementary Material 2). The classroom had a computer 
connected to the Internet and a projector so that both the 
instructors and the students could give video or PowerPoint 
presentations. There were a total of three instructors for the 
Earth science workshop: two college professors with 
doctorates and a K-12 teacher with at least one master's 
degree. 

Week 1: Climate Mechanism Jigsaw 

The first three of the essential principles of climate 
literacy recommended by the U.S. Global Change Research 
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TABLE I: Self-reported data on workshop participants and 2010-2011 Ohio Department of Education (2011c) school ratings. 


Grades Taught 

Subjects Taught 

Gender 

Completion 

School Rating 

Targeted District? 1 

1 

All 

F 

All 

Academic watch 

Yes 

1-8 

Special education 

F 

All 

Academic watch 

Yes 

3-5 

Science 

F 

Summer 

Academic watch 

Yes 

4-5 

Math & science 

F 

All 

Academic watch 

Yes 

4-6 

Special education 

F 

All 

Academic watch 

Yes 

4-8 

Math, social studies 

F 

All 

N/A (private) 

No 

5-6 

Special education 

F 

Week 1 

Not rated 2 

Yes 

6 

Science & math 

F 

All 

Excellent w/distinction 

No 

6-8 

Science 

F 

All 

Excellent 

No 

6-8 

Science 

M 

All 

Excellent 

No 

6-8 

Science 

F 

All 

Excellent 

No 

6-8 

Science 

F 

All 

Excellent 

No 

6-8 

Science 

F 

All 

Excellent 

No 

7 

Science 

M 

All 

Excellent w/distinction 

No 

7-8 

Special education 

F 

Week 1 

Effective 

Yes 

9-12 

Biology & physical science 

M 

All 

Excellent 

No 

9-12 

Biology 

F 

All 

Excellent 

No 

9-12 

Science 

M 

Summer 

Effective 

No 

10 

Biology 

M 

All 

Excellent 

No 

11-12 

Chemistry 

M 

Summer 

Continuous improvement 

Yes 


J Two high-need districts were given priority for recruitment and registration: Dayton City and Trotwood. 
2 Too few graduating students to assess since 2006; was in academic emergency in 2005-2006. 


Program (2009) were addressed during the first week (Table 
II). Five important topics for understanding climate were 
divided up among the participant groups. Each group of four 
was given a list of questions to address and instructed to 
develop and teach a lesson to their colleagues (see 
Supplementary Material 3 for scaffold). The topics were (1) 
seasons: sunlight, latitude, and tilt, (2) oceans, (3) the 
Coriolis effect: cyclones and hurricanes, (4) convection cells: 
Hadley, Ferrell, and polar, and (5) biomes (Koppen zones): 
vegetation and climate. A sixth topic, water vapor and 
humidity, was covered by the instructor because there were 


only 20 participants. The groups were to address these 
questions in a lesson. They were assigned a set of guidelines 
(the CoRe form) for developing a pedagogical content 
knowledge (PCK) framework for their topic (Mulhall et al., 
2003). The CoRe form has the teacher start by breaking a 
topic into big ideas and addressing a series of issues for each 
idea, such as why it is important, how student knowledge 
will affect their teaching of it, strategies for addressing 
students' confusion surrounding it, etc. 

Some groups incorporated limited pre- and post-testing 
(3 or 4 questions) into their lessons. Most were demonstra- 


TABLE II: Climate literacy principles addressed by the 2012 ITQ Earth Science Enhancement Workshop. 


Principle 

Primarily Addressed 

Week 

Project 

(1) The sun is the primary source of energy for Earth's 
climate system. 

1 

Climate Mechanism Jigsaw Pt. I 

(2) Climate is regulated by complex interactions among 
components of the Earth system. 

1 

Climate Mechanism Jigsaw Pt. I 

(3) Life on Earth depends on, is shaped by, and affects 
climate. 

1 & 2 

Climate Mechanism Jigsaw Pt. I, Prehistoric Zoo 

(4) Climate varies over space and time through both 
natural and man-made processes. 

2 

Prehistoric Zoo & Trends through Time 

(5) Our understanding of the climate system is improved 
through observations, theoretical studies, and modeling. 

3 

Predictions vs. Evidence 

(6) Human activities are impacting the climate system. 

3 

Predictions vs. Evidence 

(7) Climate change will have consequences for the Earth 
system and human lives. 

3 

Predictions vs. Evidence 
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FIGURE 1: Map used for the Pangea Project, modified from Blakey (2009). The information on the map legend, scale, 
and positions of rifts and trenches was communicated verbally in the 2011 workshop. 


tions with a hands-on component, but one group gave a 
lecture. Each participant was provided with a copy of Project 
Earth Science: Meteorology (NSTA, 2005), which provided 
readings, suggested activities, and links for resources, but 
after a close review, all of the participants elected to research 
on the Internet on their own to find information and ideas. 
Some developed demonstrations by experimenting with 
material in the classroom, using trial and error to determine 
the best procedure. 

The second part of the jigsaw gave participants an 
opportunity to apply their understanding of climate. The 
participants formed new groups, with all participants in the 
new groups coming from different topic groups from the 
climate-mechanism lesson that they had just completed. 
They were given a map of tropical Pangea during the mid- 
Triassic (Fig. 1). The instructor had marked six sites on it. 
Each group was assigned a site and told: "Those crazy 
teachers in the life-science workshop next door have gotten 
ahold of a time machine that will take them 230 million 
years in the past. They want to go back and establish a base 
so they can study evolution of early mammals and dinosaurs 
in the Triassic. Examine the geography of your site and 


determine the probable climate, including temperature 
ranges, prevailing winds, and seasonal rainfall. Would it be 
a good place for a research base? What about potential 
geologic and weather hazards? Will this site be safe? Will 
there be drinking water? How can the researchers generate 
electricity? Describe the local ecosystems." The participants 
were told to determine the paleolatitude of their base and to 
apply the information from the first part of the jigsaw on 
convection cells, the ocean, etc., to work out the climate 
(worksheet in Supplementary Material 4). When the 
participants presented their site analyses, most went beyond 
the questions that they had been given. Several named their 
bases. 

Week 2: Trends through Time Jigsaw 

In the second week, the activities focused on the third 
and fourth essential principles of climate literacy (U.S. 
Global Change Research Program, 2009; Table II). On 
Thursday afternoon of week 1, participants from the Earth 
science and life science workshops went on a field trip 
together to collect Ordovician fossils from Caesar's Creek 
State Park in Waynesville, Ohio. Both groups were looking 
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at problems of change over long timescales. Caesar's Creek 
fossils were left by ancient marine organisms, including horn 
corals and many types of brachiopods, providing the 
participants with spectacular evidence of environmental 
and biological changes and introducing the concept of 
geologic timescales. Over the weekend, the Earth science 
participants completed personal time-line worksheets (Teed 
and Wright, 2014) to give them a sense of the scale of 
geologic history. At the end of week 2, they visited another 
site, Oakes Quarry, where Silurian fossils from a different 
environment (particularly crinoids) had been preserved. 

For the Prehistoric Zoo exercise, the Earth science and 
life science workshop participants were combined and 
divided into new groups, each with a pair of Earth scientists 
and a pair of life scientists. Each group was assigned a 
Phanerozoic time period (e.g., the Devonian). The life 
scientists were instructed to choose charismatic extinct 
megafauna for a zoo and to describe the ecological and 
evolutionary characteristics of their choices. The Earth 
scientists were tasked with planning the expedition back in 
time to do the collecting. They were told to study the 
paleogeography, paleoclimate, and predator populations. 
The instructors recommended the Paleomap Project (Sco- 
tese, 2003), Colorado Plateau Geosystems (Blakey, 2014), 
and the University of California-Berkeley Museum of 
Paleontology (University of California, 2014) as particularly 
helpful resources, but the participants preferred to find most 
of their sources on their own. The Earth scientists would use 
their information to predict the hazards that time-traveling 
life scientists would probably face and to select a relatively 
safe location for a field station. Above all, they were not to 
allow the life scientists to get themselves killed. The life 
scientists and Earth scientists presented the plans for their 
groups to the other participants of the two workshops. Once 
again, the participants went beyond the initial requirements 
for the project (described above). One group came wearing 
appropriate garb (including a pith helmet for one member of 
the group). Another group showed pictures of their 
prospective base. A third group chose animals based on 
recently discovered fossils. 

The second part of the jigsaw was only for participants 
in the Earth science workshop. Each of these groups was 
composed of two members who had studied Paleozoic 
periods (different ones) during the first part of the jigsaw 
(the Prehistoric Zoo exercise), one from a group that had 
worked on a Mesozoic period, and one from a group that 
had studied a Cenozoic period. One group was assigned to 
search for and describe patterns in extinction over the last 
545 Ma, another to find patterns in global climate, another in 
sea level, another in tectonic activity, and the last group in 
atmospheric composition (see Supplementary Material 5 for 
advice to guide the investigation). Each group then 
presented their findings to the other participants. 

Week 3: Anthropogenic Climate Change: Predictions 
vs. Evidence 

Much of week 3 was taken up by the design of unit 
plans, but the participants had been reading a text: Dire 
Predictions (Mann and Kump, 2009) as homework for the 
previous 2 weeks. The participants started the week by 
plotting data showing how carbon dioxide concentrations 
have increased for the last two decades (Richardson, 2013). 
The final content project was to list the environmental 
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problems that people anticipate being caused by global 
warming (both in the book and in the news), to research one 
of them (per group), and to break down the information into 
two categories: (1) evidence of changes already observed, 
and (2) predictions of changes that are inferred to be 
happening or likely to happen based on scientific principles. 
The activity emphasized the last three essential principles of 
climate literacy (U.S. Global Change Research Program, 
2009; Table II). The participants were able to compile a fairly 
long list of specific problems through class discussion (see 
Supplementary Material 6). Each group collected and 
described evidence for a different problem and explained 
the reasoning that allowed people to understand how those 
problems could be caused by global warming. The topics 
that the groups studied were hurricanes, methane produc¬ 
tion, glacial retreat, wildfire, and droughts. 

During the workshop, several of the participants 
paraphrased the guiding principle from Climate Literacy 
(U.S. Global Change Research Program, 2009, p. 6): 
"Humans can take actions to reduce climate change and 
its impacts." However, there was not enough time to address 
the future of climate change using an inquiry approach, 
although the physical science workshop taking place across 
the hall spent over a week on item "E" of the guiding 
principle, dealing with energy efficiency and alternative 
energy. 

Unit-Plan Project 

During the summer workshop, each participant also 
developed lesson plans (generally whole units) to allow 
them to use their updated content knowledge and their 
recent experience with inquiry in their own classrooms 
during the following school year. Throughout the workshop, 
the participants practiced inquiry learning techniques, 
particularly jigsaw. During the first week, the participants 
became familiar with the CoRe form (Mulhall et al., 2003), 
which they used to pull together content objectives for a 
simple lesson plan to address their part of the weather/ 
climate jigsaw. 

During the second week, each participants chose a unit 
topic and completed a new CoRe form for the unit. Of the 14 
lesson-plan units that were taught and assessed during the 
follow-up, 11 addressed topics from week 1: basic climate 
and weather mechanisms, seasons, and biomes (Table III). 
All of the units extended beyond the basic demonstration 
used in week 1. One lesson covered fossils using material 
collected from the field trip and planned expeditions back in 
time similar to the project for week 2. The other two dealt 
with climate change over geologic and recent history, 
covered in week 2 and week 3, one using a trends-in-time 
scaffold like the one developed for this workshop. The 
popularity of week 1 topics for units indicates that the 
participants need more opportunities to design short lessons 
during the summer workshop that they can then build into 
units as they did in week 1. So perhaps weeks 2 and 3 should 
include some minilesson design. 

The participants finished written descriptions of the unit 
plan and ordered supplies to run it by the end of the third 
week. Some participants worked in teams, each writing a 
part of the lesson plan, but committing to run the whole 
unit, whether together or in separate classrooms. The 
written description was graded based on completeness. 
The unit had to contain a description of the lessons, 
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TABLE III: Pre- and post-test results of follow-up on participants' K-12 students. Overall values for <g> were weighted by the 
number of students tested for each unit. 


Grade 

District 

Class 

Topic 

No. 

Students 

Mean 
Pretest (%) 

Mean 

Post-Test (%) 

Mean 
<g> (%) 

6 

Bethlehem Lutheran 

Social studies 

Monsoons 

5 

56 

78 

50 

7 

Centerville City 

Science 

Climate 

50 

9 

62 

58 

7 

Centerville City 

Science 

Climate 

48 

5 

51 

48 

7 

Centerville City 

Science 

Climate 

49 

7 

68 

66 

7 

Centerville City 

Science 

Climate 

48 

8 

52 

48 

7 

Centerville City 

Science 

Climate 

49 

8 

65 

62 

1 

Dayton City 

All subjects 

Seasons 

25 

56 

84 

60 

7-8 

Dayton City 

Special ed. 

Biomes 

17 

23 

46 

30 

5 

Dayton City 

Science 

Seasons 

45 

2 

54 

53 

6 

Oakwood City 

Science 

Fossils 

21 

21 

90 

88 

7 

Oakwood City 

Science 

Climate 

40 

7.1 

50 

46 

11-12 

Troy City 

Biology II 

Climate Change 

40 

40 

83 

72 

10 

Vandalia-Butler City 

Biology 

Climate Change 

20 

43 

52 

16 

9 

Yellow Springs 

Earth/space science 

Biomes 

19 

14 

67 

62 

Overall 

476 



56 


assessments, and resource bibliographies used. The unit had 
to be centered around a finished CoRe framework and 
include a brief description of the participant's school and 
students, relevant elements from the Ohio standards 
(benchmarks from the 2002 standards, content statements 
for the 2011 standards), a content outline (similar to the one 
in Supplementary Material 1), and a pedagogical discussion 
addressing inquiry and the learning cycle. Finally, one of the 
assessments for the unit had to be a content test that the 
participants would use as a pre- and post-test to assess the 
impact of the units on their own students. 

Follow-Up Program 

The follow-up program extended over the 2011-2012 
school year. During this time, the participants finished, 
taught, and tested the unit plans that they had developed 
during the workshop. Participants also returned to Wright 
State University for three short (half-day to full-day) 
workshops. In August 2011, they presented their unit plans 
to one another and critiqued each other's pre- and post-test 
questions, a form of basic validation. During this meeting, 
they also picked up the supplies that they had ordered for 
their units and made appointments with one of the master 
teachers who had helped teach the summer workshop to 
visit their classrooms at least twice. Between September 2011 
and May 2012, the master teacher helped the participants 
teach their unit plans and administer their pre- and post¬ 
tests. 

In October 2011, the participants had rock and mineral 
identification training, since many of them had expressed 
interest in this topic during the summer workshop. The 
instructor also reviewed the requirements for analyzing the 
pre- and post-test data from the unit plans for the summary 
that would be due in May. The participants were required to 
create a spreadsheet with the response to each test question 
from each student (numbered, not named) and calculate the 
pretest average, post-test average, and normalized gain 


(Hake, 1998). The participants also had to write a reflection 
on their experience of teaching their lesson and an 
interpretation of the pre and post-test results. Finally, in 
May 2012, the participants met to present their students' 
pre- and post-test results from their lesson plans and to 
report on what worked as expected and what did not. They 
turned in the papers and spreadsheets with their results. 

WORKSHOP RESULTS 

Impact of Summer Workshop on Content Learning of 
Participants 

The first day of the summer workshop, participants took 
a pretest to assess prior content knowledge. It consisted of 
12 questions to be answered with a few sentences (Table IV) 
and asked the participants to use mechanistic (cause and 
effect) explanations rather than technical terms whenever 
possible. The participants took the same test again on the 
afternoon of the last day of the workshop, and the results of 
the postworkshop test were compared to the results from the 
first to estimate content-knowledge gain during the work¬ 
shop. 

The participant content-knowledge test used a short- 
answer format rather than multiple choice so that partially 
correct answers could be scored as such. On a typical 
multiple-choice test, incomplete understanding requires 
students to make educated guesses, which appear to be 
either completely wrong or completely right after scoring. 
The short-answer format provided a more accurate reflection 
of participant content knowledge. 

The workshop content pre- and post-test questions 
were based on the content outline (Supplementary Material 
1), four questions from each week of the course. The 
questions and most of the scoring rubric were written by 
Rebecca Teed and validated in the sense of being reviewed 
by the other two instructors of the workshop and by the 
external evaluator. There was no opportunity to pilot-test the 
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TABLE IV: Content pre- and post-test questions for the Earth science summer workshop participants. The test form also included 
a space for the participant name and the instructions: "Use mechanistic (cause & effect) explanations rather than technical terms 
whenever possible." 


(1) What kind of weather do you associate with low air pressure? 

(2) What is one major difference between the climate of Dayton, OH, and that of Minneapolis, MN? Furthermore, what causes that 
difference? 

(3) Where are most of Earth's large deserts? Why? 

(4) What is an isobar, and why is it important? 

(5) What is the major cause of sea-level change over the last 500 million years? 

(6) What are two pieces of evidence for sea-level change from the rock record? 

(7) Why do we expect climate to get warmer with increased levels of atmospheric carbon dioxide? 

(8) Name one ancient time period, and compare the global climate at that time with the present. 

(9) What are two pieces of evidence for or against anthropogenic global warming? 

(10) What are two potentially serious effects of global warming? 

(11) What is remarkable about the twentieth-century record of carbon-dioxide buildup (recorded by the Keeling/Mauna Loa curve)? 

(12) Water vapor is a more powerful greenhouse gas than carbon dioxide, and often more plentiful. Why aren't any politicians 
focused on controlling it? 


questions before the workshop. While the pretest was being 
marked, the scoring rubric (Supplementary Material 7) was 
expanded to include several unanticipated partially correct 
participant answers. 

The average pretest score was 38.1%, and the average 
post-test score was 75.6%. Every participant's total score 
increased (Fig. 2), and the class' total score on each question 
increased between the pre- and post-test (Fig. 3). It is 
statistically less likely for participants with higher pretest 
scores to increase those scores on the post-test, because they 
have fewer pretest questions to answer correctly (or more 


completely correctly) than participants with lower pre-test 
scores. So score gains were normalized to account for head- 
room on the pretest using the following formula from Hake 
(1998): 

<g> = (post-test score — pretest score)/(100% — pretest 
score). 

The average normalized gain <g> was 63.8% for the 
workshop content pre- and post-test, which is higher than 
average compared to similar pre- and post-tests for the 13 
ITQ workshops offered from 2006 to 2010 (Basista, pers. 
comm., July 12, 2011). 



FIGURE 2: Participant content pre- and post-test scores from the summer workshop. 
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Question # 


FIGURE 3: Scores on individual content pre- and post-test questions from the summer workshop. 


Cronbach's alpha, a measure of consistency among 
responses to test items, is commonly used as a measure of 
reliability on tests (Cronbach, 1951). In this case, it is an 
estimate of whether a total score for answers to the various 
questions that make up the test measures the same 
composite idea, such as understanding of climate. The alpha 
for the pretest was 85%, and for the post-test, it was 73%, 
highly to moderately reliable in a research context (Murphy 
and Davidshofer, 2005), but the sample size (18 participants) 
is small enough that these alpha values must be treated as 
estimates. 

There was a lot of variation among the participants with 
respect to prior knowledge, but some of the difficulty with 
questions like #7 on the pretest may have been due to the 
requirement of a mechanistic explanation. Many participants 
answered question #7 ("Why do we expect climate to get 
warmer with increased levels of atmospheric carbon 
dioxide?") with "CCb is a greenhouse gas," on the pretest 
without explaining what a greenhouse gas is. In the 
workshop, the instructors and participants focused on 
cause-and-effect explanations for climate change, especially 
for classroom instruction. On the post-test, most participants 
gave at least a partial explanation of how CO 2 warms the 
atmosphere. 

Question 12 about why more politicians want to control 
CO 2 rather than water vapor was intended to be difficult 
because it requires an understanding of feedbacks. Partici¬ 
pants with less experience with systems thinking were not 
expected to develop such an understanding during the 
workshop, but those with strong backgrounds in fields like 
ecology were able to improve their answers to question 12 
between the pre- and post-test. Respondents who men¬ 
tioned politics did not include a scientific explanation in their 
answers, and it appears that these were fallback responses. 


Scores for this question were highly correlated with scores 
on other questions. 

Impact of Unit Plans on Content Learning of K-12 
Students During the Follow-Up Program 

The participants included pre- and post-tests in their 
unit plans in order to assess their students' content- 
knowledge gains. The number of questions included in their 
tests ranged from six to 15 questions. The shorter tests were 
short-answer tests, graded on a continuous scale, with 
points for partially correct answers, whereas the longer tests 
were multiple choice. During the August follow-up meeting, 
the participants gave presentations in which they briefly 
described their units and then reviewed each of their pretest 
questions. The instructor and the other participants recom¬ 
mended edits to many of these questions. The edits were 
informative because some of the participants had done a 
similar follow-up program in life-science ITQ workshops in 
previous years and had experience with developing and 
administering pre- and post-tests. 

During the follow-up program, the participants taught 
the units that they had developed, administering the pretest 
at the start of the first class and the post-test at the end of 
the last class. The average normalized learning gain <g> for 
the 476 grade 1-12 students who participated in the 14 units 
and were assessed was 56% (Table III). Results were 
excluded from one class in which the students collaborated 
on the responses to the pretest. The greatest gains were in a 
highly motivated class (high-performing middle-school 
students studying fossils and planning their own time-travel 
expeditions). The lowest gains were reported by a participant 
who used direct instruction rather than inquiry to address 
the content covered by several pre-/post-test questions. That 
participant described limited student participation in the 
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TABLE V: Change in percentage of participants responding "strongly agree" on the Ohio Board of Regents pre- and post-survey. 


Statement 

% Strongly Agree 

Pre- 

Post- 

I have a good understanding of fundamental core content in my discipline. 

60 

87 

I have a good understanding of relating classroom activities to Ohio's academic standards 

40 

73 

I have a good understanding of how to assess student learning in multiple ways. 

20 

73 

I have a good understanding of effective questioning techniques and their use in the classroom. 

33 

73 

I have a good understanding of how to use technology effectively in the classroom. 

27 

40 

I have a good understanding of the methods necessary to teach math and/or science concepts effectively. 

33 

87 

I believe I am an effective teacher. 

47 

73 

I am excited about teaching in my subject area. 

80 

87 

I am interested in networking with teachers and other professionals. 

73 

87 


inquiry activities. Students in a special-education classroom, 
where much of the time in every unit was spent on reading 
skills (which were not directly assessed in the pre- and post¬ 
test), showed an average normalized gain of 30%. 

Impact on Attitude in Participants 

Participants provided feedback regarding the impact of 
the institute through two methods: an Ohio Board of 
Regents (OBR) pre-post survey and a Horizon Local 
Systemic Change (LSC) pre-post survey (Horizon, Inc., 
2006). The OBR survey is provided by OBR for all Ohio ITQ 
recipients. Participants completed both surveys on the first 
day of the summer institute and during the final follow-up 
workshop approximately 12 mo later (n = 15). There are no 
data regarding the reliability or validity of the OBR 
instrument. 

The OBR survey includes several Likert-scale questions 
asking the participant to indicate the level of agreement, 
ranging from strongly agree (1) to strongly disagree (5). The 
statements are written in a manner that reflects strong 
teaching attitudes, beliefs, and practices. The responses to 
the questions in #12 (participants' attitudes, beliefs, and 
degree of confidence) indicate that more participants 
strongly agreed with the positive statements about their 
preparedness and attitudes towards teaching at the end of 
the follow-up program (Table V). The largest increases are 
indicated for teaching methods and assessment. At the end 
of the follow-up program, more participants felt strongly 
that they were knowledgeable and had more understanding 
of their content and pedagogy and that they were effective 
teachers. 

The OBR survey also included seven paired statements 
representing a continuum of practice on which participants 
placed themselves. One such paired statement asked the 
participant to indicate how students generally work in the 
participant's classroom. The five-point continuum provided 
was: Students generally work in groups cooperatively (1) to 
Students generally work independently (5). Topics covered 
in the paired statements included assessment practices, 
motivation efforts, classroom interaction practices, instruc¬ 
tional practices, and student roles. The paired-statements 
responses indicated movement in one pair regarding 
instructional activities in classrooms. The paired statement 
was: "Students generally learn concepts and processes using 
hands-on approaches." After the follow-up program, 


participants indicated that they had moved on the contin¬ 
uum to more hands-on approaches in their instruction. 
Responses to other OBR questions indicate participants had 
increased their level of involvement in sharing their teaching 
with colleagues and other professionals. At the end of the 
summer workshop and again at the end of the follow-up 
program, all participants indicated they would recommend 
the institute to their colleagues. 

The LSC teacher questionnaires were initially developed 
in 2000 through an National Science Foundation-funded 
contract with Horizon Research Incorporated. The intent 
was to develop instrumentation to track systemic change in 
teachers' attitudes and perceptions regarding their mathe¬ 
matics and/or science content preparedness, pedagogical 
preparedness, classroom practices, and support of their 
school principals for math and science teaching. Expert 
reviews established the validity of the instrument. The 
questionnaire contains 29 questions, all of which have from 
four to 24 subquestions. Respondents have a choice of five 
Likert-style choices of strongly disagree to strongly agree; or 
four Likert choices of not adequately prepared to very well 
prepared; or four Likert choices of not important to very 
important. The items on the questionnaire were combined 
into composite variables through factor analysis to provide 
more reliable estimates of teachers' preparedness and 
classroom practices (Germuth et al., 2003). The composites 
of interest for this study were attitudes towards teaching, 
pedagogical preparedness, and principals' support. Germuth 
et al. (2003) used Cronbach's alpha to measure internal 
reliability estimates for these composites, and the values 
(0.67-0.76, p < 0.05) are in the acceptable to moderate range 
(Murphy and Davidshofer, 2005). 

A Wilcoxon rank sum test using total scores indicated 
significant gains in attitudes towards teaching, W = — 74, p = 
0.01 (pre-n = 15, post-n = 15); in addition, the increase in 
pedagogical preparedness was significant, W = —120, p = 
0.0007 (pre-« = 15, post-n = 15). The Wilcoxon signed rank 
test is a nonparametric alternative to the two-sample t-test 
and is preferred when the population cannot be assumed to 
be normal. The sample for this ITQ project cannot be 
reasonably assumed to be normal. There was no significant 
change in principals' support. The LSC survey results are 
supported by the OBR survey results. 

In an open-ended request for program impact, one 
teacher wrote: 















596 R. Teed and S. Franco 


J. Geosci. Educ. 62, 587-597 (2014) 


"I am ten times the teacher I was before ITQ! Being 1-8 
certified didn't mean I had the content to be effective up to 8, 
just certified—now I have both. The best take aways were the 
actual hands-on activities to use in class—and an 
understanding of the how and why for each activity." 


DISCUSSION 

Assessment Limitations 

It is not possible to compare the workshop's effect on 
participant content knowledge and attitudes to those of the 
continuing-education programs within local school districts 
as those programs either do not collect or do not release pre- 
and post-test results. Similarly, the participants' students 
clearly increased their knowledge of the topics covered in the 
units that the participants taught and assessed, but there is 
no control group data from classes taught by teachers who 
did not participate in the program or by those who dropped 
out. Even before the workshop and follow-up program, the 
participants may have been excellent and confident teachers, 
and their lessons were probably fairly effective before. Many 
of them had participated in similar workshops in previous 
years. 

The motivation of the participants was critical to the 
success of the workshop and the follow-up program. The 
teachers who completed the program devoted considerable 
effort outside of class by reading the textbook and designing 
lessons. The participants who dropped out of the summer 
program did not turn in homework or turned in unaccept¬ 
able work. However, since they did not take the post-test or 
post-survey, there is no way to measure what effect the 
program would have had on them. 

Workshop Recommendations 

The topics of ITQ workshops change during each 
offering to allow participants to return to study content 
and design curriculum appropriate to different standards 
each year. If there were funding for another Earth Science 
workshop, the next topic would be plate tectonics. However, 
some of the techniques, particularly the first part of the 
jigsaw in week 1, with a set of scaffolding questions (perhaps 
not as detailed as those in Supplementary Material 3), CoRe 
exercise (Mulhall et al., 2003), and minilesson, could be 
adapted to the new content. Since many of the week 1 
minilessons inspired the participants to create and design 
unit plans on those topics, the same format should probably 
be repeated in week 3, using week 3 topics. The final content 
project, which required the participants to discuss the use of 
evidence and reasoning, could be used as the scaffold for any 
Earth science topic. If the participants invested an afternoon 
of what would have been project time into a second set of 
minilessons, more of them might consider week 2 and week 
3 topics for their unit plans. 

One problem with working with high-needs districts 
was recruitment; another was retention. Teachers from 
Dayton City schools mentioned that they could participate 
in no-cost professional development programs that do not 
take time beyond their usual work hours. Of the eight 
participants from partner districts, four did not complete the 
program. Programs like ITQ require strong support from 
school and district administrators. The ITQ participant 
application required a principal's signature. However, one 


of the participants who dropped out of the follow-up 
program did so because her principal forbade her not only 
to teach the lesson-plan unit she designed in the workshop 
but to teach any science at all. She was told to have her 
students practice math problems during the science period 
in order to improve their standardized test scores. The 
original partnership letter written by one of the school 
districts from which most of the dropouts came made no 
specific commitment to require their teachers to finish the 
program. Similar workshop programs described in the 
literature, such as the Satellites, Weather and Climate 
Teacher Professional Development Program at the Univer¬ 
sity of Vermont (Dupigny-Giroux et al., 2012), have also 
struggled to recruit enough participants despite having 
measured positive effects on the content knowledge and 
pedagogical preparedness of the teachers who were in¬ 
volved. 


CONCLUSION 

The 2011-2012 Earth science ITQ program for in-service 
teachers focused on climate change over geologic time. The 
participants engaged in jigsaw exercises to increase their 
content knowledge and developed multilesson unit plans 
that they taught in their own classrooms during the 
following year. The participants' pre- and post-test results 
indicate that the summer workshop increased teacher 
content knowledge. Their attitudes towards teaching and 
pedagogical preparedness significantly improved over the 12 
mo period of the workshop and follow-up program. Though 
there were numerous influences that could have contributed 
to the changes documented here, several of the participants 
stated that the professional development experience was one 
of the dominant contributors. 
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